1. Field of the Invention
The present invention relates to a user interface technique, in particular to a technique of presenting a sense of force in the fields of virtual reality and mixed reality.
2. Description of the Related Art
As a kind of tactile sense presentation device, force sense presentation devices are known, which present a sense of force by stimulating deep sensations in humans.
In a known typical example of the force sense presentation device, a user grips a stylus pen at the end of a robot arm. The motor of the robot arm controls the output, thereby presenting a sense of force to the hand gripping the stylus pen (T. M. Massie, J. K. Salisbury, “The PHANTOM Haptic Interface: A Device for Probing Virtual Objects”, ASME Haptic Interface for Virtual Environment and Teleoperator Systems 1994, In Dynamic Systems and Control, Vol. 1, pp. 295-301 (to be referred to as “Massie et al.” hereinafter)). Another device is known which controls the tensile forces of several strings attached on a ball or fingerstalls, thereby presenting a sense of force to the hand gripping the ball or the fingers inserted into the fingerstalls (Makoto Satoh, Yukihiro Hirata, and Hiroshi Kawarada, “Proposals of Space Interface Device SPIDAR”, IEICE Transactions D-II Vol. J74-D-II, No. 7, pp. 887-894, July (to be referred to as “Satoh et al.” hereinafter)). Still another device is known which has a handle on a floater that is levitating magnetically and causes a user to grip the handle. The magnetic force between the stator and the floater is controlled to present a sense of force to the hand gripping the handle (P. J. Berkelman, Z. J. Butler, and R. L. Hollis, Design of Hemispherical Magnetic Levitation Haptic Interface Device”, 1996 ASME International Mechanical Engineering Congress and Exposition, Atlanta, November 1996, DSC-Vol. 58, pp. 483-488). Still another technique measures the position of a human hand by using an LED. Upon detecting the human hand's approach to an object, a plate attached to a robot arm encounters the hand (Y. Yokokohji, R. L. Hollis, and T. Kanabe, “What You can See Is What You can Feel—Development of a Visual/Haptic Interface to Virtual Environment—”, In Proceedings of 1996 IEEE Virtual Reality Annual International Symposium (VRAIS '96), pp. 46-53 (1996)). All devices above are ground based force sense presentation devices.
On the other hand, a body based force sense presentation device which has a ground point on a body and is called an exoskeleton type is known (U.S. Pat. No. 5,184,319). This exoskeleton type force sense presentation device fixes a wrist as a support point of force sense presentation. Driving media such as wires are arranged along the exoskeleton and driven to present, to the fingertips, a sense of force in gripping a virtual object.
A body based arrangement which is not of the exoskeleton type is also known (Japanese Patent Laid-Open No. 2002-304246). This device causes a user to grip a force sense presentation unit. A link mechanism is provided between the force sense presentation unit and a unit fixed to a wrist. An actuator arranged on the fixed unit presents a sense of force to the hand gripping the force sense presentation unit via the link mechanism.
This arrangement will be described with reference to FIG. 18. FIG. 18 is a view showing the arrangement of a prior art disclosed in Japanese Patent Laid-Open No. 2002-304246. This body based force sense presentation device causes a user to grip a grip unit 300 to present a sense of force. Support units 200 fix the wrist. Four variable length connection units 400 made of wires are provided between the grip unit 300 and the support units 200. The grip unit 300 is pushed, pulled, or rotated relative to the support units 200, thereby presenting a sense of force to the gripping hand. In addition, the position of the grip unit 300 is detected by measuring the lengths of the variable length connection units 400, thereby determining interference with a virtual object and calculating its insertion depth and speed. A presentation force is obtained based on the calculated insertion depth and speed, and the variable length connection units 400 are driven. The prior art also mentions an application which presents a sense of force in hitting a virtual object (ball) in the virtual environment, making the grip unit 300 gripped by the user simulate the grip of a tool (e.g., bat, golf club, and tennis racket) used in the real environment.
Still another arrangement is also known which causes a user to grip a force sense presentation unit and presents a sense of force to the hand gripping it (N. Nakamura, Y. Fukui, “Development of a Force and Torque Hybrid Display “GyroCubeStick””, world HAPTICS 2005, Pisa, Italy, Mar. 18-20, 2005, pp. 633-634, Japanese Patent Laid-Open No. 2005-190465). FIGS. 17A and 17B show the arrangement of this prior art. Referring to FIG. 17A, a grip unit 1000 to be gripped by a user has a tubular shape. Two eccentric rotors 1100 are placed coaxially on the rotation axis. FIGS. 17B, 17C, and 17D show the phase relationship between the eccentric rotors 1100. FIG. 17B shows the two eccentric rotors 1100 which are in phase and rotate in the same direction, thereby presenting vibration to the hand gripping the grip unit 1000. FIG. 17C shows the two eccentric rotors 1100 which have a fixed phase difference of 180° and rotate in the same direction. When the eccentric rotors 1100 rotate, they can present a torque to the hand gripping the grip unit in accordance with their angular velocities. FIG. 17D shows an arrangement capable of presenting a sense of force by rotating the eccentric rotors 1100 in opposite directions and controlling acceleration and deceleration.
However, the above-described prior arts have the following problems. Of the prior arts, the ground based force sense presentation devices disclosed in the references have a support point for force sense presentation or a force sense presentation mechanism outside the human body. The manipulation area is limited, and its narrowness poses a problem. Interference between the human and the robot arm or strings also impairs the reality.
The body based force sense presentation devices including the force sense presentation device of the exoskeleton type have a fixed portion serving as a support point on part of the human body. This solves the problem of narrow manipulation area in the ground based devices. However, since the support point for force sense presentation is provided on a body part such as a wrist, attachment/detachment is cumbersome, resulting in a sense of incongruity in mounting.
The force sense presentation device described with reference to FIG. 18 can present a sense of gripping a real tool by making the grip unit gripped by the user simulate the grip of a tool (e.g., bat, golf club, and tennis racket) used in the real environment. Additionally, since the wrist serves as the support point, cumbersomeness in attachment/detachment decreases. However, since the support point exists only on the wrist, the reaction of the force that acts to present the sense of force readily concentrates to the wrist, resulting in a larger sense of incongruity. The variable length connection units (wires) hinders a task requiring two-hand operation, and the other hand cannot grip the grip unit without the support point. That is, this force sense presentation device is unsuitable for a task requiring two-handed operation.
The force sense presentation device described with reference to FIGS. 17A to 17D can solve the problem of narrow manipulation area in the ground based force sense presentation devices. This device can also solve the cumbersomeness in attachment/detachment and the sense of incongruity in mounting in the body based force sense presentation device. However, the force sense presentable by the system shown in FIGS. 17A to 17D uses the human sensory characteristic and illusion. The actual force generation is intermittent. Since the user perceives the sense of vibration, the reality is impaired. Furthermore, the perceptible sense of force is small. In continuous use, the user gets used to it, and force sense perception further weakens.
FIG. 43 is a block diagram showing the functional arrangement of a conventional system for implementing force sense presentation in virtual reality (Kenneth Salisbury, Francois Conti, and Federico Barbagli, “Haptic Rendering: Introductory Concepts”, IEEE Computer Graphics and Applications, January/February 2004, pp. 24-32).
Referring to FIG. 43, reference numeral 1 denotes a force sense presentation device. A typical example of the force sense presentation device 1 controls a robot arm with a stylus pen at the end. The motor of the robot arm controls the output, thereby presenting a sense of force to the hand gripping the stylus pen (Massie et al).
Another device controls the tensile forces of several strings attached on a ball or fingerstalls, thereby presenting a sense of force to the hand gripping the ball or the fingers inserted into the fingerstalls (Satoh et al).
There is also a device called an exoskeleton type (U.S. Pat. No. 5,184,319). This exoskeleton type force sense presentation device fixes a wrist as a support point of force sense presentation. Driving media such as wires are arranged along the exoskeleton and driven to present, to fingertips, a sense of force in gripping a virtual object.
A force sense rendering unit 20 includes a interference determination unit 210, force sense calculation unit 220, and control unit 230.
The interference determination unit 210 acquires, from the force sense presentation device 1, a position X of a virtual object (avatar) that simulates a machine tool as a manipulation target of the user and obtains an interference state by using the acquired position and the position of the object as the manipulation target of the machine tool. The interference state is an insertion depth S representing the degree of insertion of the avatar in the object.
The force sense calculation unit 220 calculates a force Fd acting on the avatar based on the insertion depth S.
The control unit 230 obtains a control value Fr to cause the force sense presentation device 1 to present the force Fd to the user and sends the obtained control value Fr to the force sense presentation device 1.
A simulation unit 30 has a simulation engine 310. The simulation unit 30 simulates deformation or movement of the object based on the force Fd calculated by the force sense calculation unit 220. The simulation result is sent to the interference determination unit 210 immediately. The interference determination unit 210 determines the interference between the object and the avatar according to the simulation result. The simulation result is also sent to a graphics engine 410 on the succeeding stage.
An image rendering unit 40 has the graphics engine 410. The graphics engine 410 generates an avatar image and an object image. The object moves or deforms based on the simulation result.
An image display unit 50 including a flat panel display or head mounted display (HMD) with a CRT or liquid crystal display panel displays an image generated by the graphics engine 410.
To present a sense of force of tightening a screw by the above-described conventional force sense presentation method, however, it is necessary to determine the interference between the thread ridge and root of an external thread and those of an internal thread, and the calculation is complex. Additionally, the reaction force and friction at multiple points need to be calculated. This makes the calculation more complex and largely increases the amount of calculation. Hence, real-time presentation or smooth and natural force sense presentation is impossible.